Spiders and Crawlers and Bots, Oh My: The Economic Efficiency and Public Policy of Contracts that Restrict Data Collection

Recent trends reveal the search by companies for a legal hook to prevent the undesired and unauthorized copying of information posted on websites. This quest has sparked a fundamental controversy over the ownership of information and the Internet itself. Some believe that information published on websites should be used only in a manner sanctioned by the publisher, so that " businesses can proceed without fear of unwanted trespassers who will steal or profit from the fruits of [the business'] labor.'' 1 Many argue, however, that the efficient exchange of factual information, unha mpered by any legal or technological barrier, unquestionably benefits society and weighs strongly against the enforceability of any restrictive mechanism. 2 Especially troublesome to those who struggle against this unwanted copying of website information are software robots, small programs that automatically and rapidly search, copy and retrieve information from websites. 3 Prevalent and controversial among software robots are shopbots or pricebots, which comb through commercial websites, A robot is a program that navigates the World Wide Web's hypertext structure, recursively retrieving documents. Robots are also referred to as " bots, " " spiders, " and " web-crawlers. " Search engines illustrate a fundamental use of robots; these robots traverse the Internet, recursively following hyperlink after hyperlink and indexing the web pages at the end of the links. The indexed web pages are then stored in the search engine's database so that a user may later query the search engine to discover the indexed page. While robots are not intrinsically harmful to the internet, as in a properly functioning search engine robot, a poorly implemented robot or a flood of identical robots can tax the Internet's infrastructure. Different than robots, but also relevant to the struggle against illegitimate copying are unauthorized deep links. Hyperlinks are software instructions sent to a networked computer to retrieve data. With websites, hyperlinks are the text and pictures, which when clicked, take the clicker to a different web-page. Deep links take the user to an interior page of another's website. Therefore, if a website wishes for all its users to enter through a homepage, an unauthorized deep link may thwart this policy. 2 extracting pricing and product information. 4 Typically, shopbots are used in conjunction with a metasite, a website that displays prices from a variety of vendors for an identical item. 5 While shopbots and metasites tend to revolve around pricing information, these …